{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qZhIHQQ1sYC3"
   },
   "source": [
    "# TensorFlow Estimators Deep Dive\n",
    "\n",
    "The purporse of this tutorial is to explain the details of  how to create a premade TensorFlow estimator, how trainining and evaluation work with different configurations, and how the model is exported for serving. The tutorial covers the following points:\n",
    "\n",
    "1. Implementing **Input function** with tf.data APIs.\n",
    "1. Creating **Feature columns**.\n",
    "1. Creating a **Wide and Deep** model with a premade estimator.\n",
    "4. Configuring **Train and evaluate** parameters.\n",
    "5. **Exporting** trained models for **serving**.\n",
    "6. Implementing **Early stopping**.\n",
    "7. Distribution Strategy for **multi-GPUs**.\n",
    "8. **Extending** premade estimators.\n",
    "9. Adaptive **learning rate**.\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/sme_academy/01_tf_estimator_deepdive.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img valign=\"middle\" src=\"images/tf-layers.jpeg\" width=\"400\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DRHI7BAm3xNn"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "  COLAB = True\n",
    "  from google.colab import auth\n",
    "  auth.authenticate_user()\n",
    "except:\n",
    "    pass\n",
    "\n",
    "RANDOM_SEED = 19831006"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "e1RwNw9Rr7BP",
    "outputId": "703f1591-741d-416d-c45b-8acc9765e44b"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import math\n",
    "import multiprocessing\n",
    "import pandas as pd\n",
    "from datetime import datetime\n",
    "\n",
    "import tensorflow as tf\n",
    "print \"TensorFlow : {}\".format(tf.__version__)\n",
    "\n",
    "tf.enable_eager_execution()\n",
    "print \"Eager Execution Enabled: {}\".format(tf.executing_eagerly())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "S0C3sEMX1nuo"
   },
   "source": [
    "## Download Data\n",
    "\n",
    "### UCI Adult Dataset: https://archive.ics.uci.edu/ml/datasets/adult\n",
    "Predict whether income exceeds $50K/yr based on census data. Also known as \"Census Income\" dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "9_FhrUo81gxq",
    "outputId": "32a35c81-580c-4c77-8cb6-cd9ab0bc1640"
   },
   "outputs": [],
   "source": [
    "DATA_DIR='data'\n",
    "!mkdir $DATA_DIR\n",
    "!gcloud storage cp gs://cloud-samples-data/ml-engine/census/data/adult.data.csv $DATA_DIR\n",    "!gcloud storage cp gs://cloud-samples-data/ml-engine/census/data/adult.test.csv $DATA_DIR"   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KZSRX6hl1tTM"
   },
   "outputs": [],
   "source": [
    "TRAIN_DATA_FILE = os.path.join(DATA_DIR, 'adult.data.csv')\n",
    "EVAL_DATA_FILE = os.path.join(DATA_DIR, 'adult.test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "sjNKmcwb1yj4",
    "outputId": "330b30e4-f1c0-4c5b-af77-aef0f54cd659"
   },
   "outputs": [],
   "source": [
    "!wc -l $TRAIN_DATA_FILE\n",
    "!wc -l $EVAL_DATA_FILE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M9q54o1g2fIF"
   },
   "source": [
    "The **training** data includes **32,561** records, while the **evaluation** data includes **16,278** records. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "Hgee9x2m16m9",
    "outputId": "306db090-1c1d-47ed-e194-ea9ce1ffc218"
   },
   "outputs": [],
   "source": [
    "HEADER = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',\n",
    "               'marital_status', 'occupation', 'relationship', 'race', 'gender',\n",
    "               'capital_gain', 'capital_loss', 'hours_per_week',\n",
    "               'native_country', 'income_bracket']\n",
    "               \n",
    "pd.read_csv(TRAIN_DATA_FILE, names=HEADER).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3dYrN-U34mjk"
   },
   "source": [
    "## Dataset Metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EeMS4rF14Zku"
   },
   "outputs": [],
   "source": [
    "HEADER = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',\n",
    "               'marital_status', 'occupation', 'relationship', 'race', 'gender',\n",
    "               'capital_gain', 'capital_loss', 'hours_per_week',\n",
    "               'native_country', 'income_bracket']\n",
    "\n",
    "HEADER_DEFAULTS = [[0], [''], [0], [''], [0], [''], [''], [''], [''], [''],\n",
    "                       [0], [0], [0], [''], ['']]\n",
    "\n",
    "NUMERIC_FEATURE_NAMES = ['age', 'education_num', 'capital_gain', 'capital_loss', 'hours_per_week']\n",
    "CATEGORICAL_FEATURE_WITH_VOCABULARY = {\n",
    "  'workclass': ['State-gov', 'Self-emp-not-inc', 'Private', 'Federal-gov', 'Local-gov', '?', 'Self-emp-inc', 'Without-pay', 'Never-worked'], \n",
    "  'relationship': ['Not-in-family', 'Husband', 'Wife', 'Own-child', 'Unmarried', 'Other-relative'], \n",
    "  'gender': [' Male', 'Female'], 'marital_status': [' Never-married', 'Married-civ-spouse', 'Divorced', 'Married-spouse-absent', 'Separated', 'Married-AF-spouse', 'Widowed'], \n",
    "  'race': [' White', 'Black', 'Asian-Pac-Islander', 'Amer-Indian-Eskimo', 'Other'], \n",
    "  'education': ['Bachelors', 'HS-grad', '11th', 'Masters', '9th', 'Some-college', 'Assoc-acdm', 'Assoc-voc', '7th-8th', 'Doctorate', 'Prof-school', '5th-6th', '10th', '1st-4th', 'Preschool', '12th'], \n",
    "}\n",
    "\n",
    "CATEGORICAL_FEATURE_WITH_HASH_BUCKETS = {\n",
    "  'native_country': 60,\n",
    "  'occupation': 20\n",
    "}\n",
    "\n",
    "FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_WITH_VOCABULARY.keys() + CATEGORICAL_FEATURE_WITH_HASH_BUCKETS.keys()\n",
    "TARGET_NAME = 'income_bracket'\n",
    "TARGET_LABELS = [' <=50K', ' >50K']\n",
    "WEIGHT_COLUMN_NAME = 'fnlwgt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bJk8Bryd6uXu"
   },
   "source": [
    "## 1. Data Input Function\n",
    "* Use [tf.data.Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) APIs: **list_files()**, **skip()**, **map()**, **filter()**, **batch()**, **shuffle()**, **repeat()**, **prefetch()**, **cache()**, etc.\n",
    "* Use [tf.data.experimental.make_csv_dataset](https://www.tensorflow.org/api_docs/python/tf/data/experimental/make_csv_dataset) to read and parse CSV data files.\n",
    "* Use [tf.data.experimental.make_batched_features_dataset](https://www.tensorflow.org/api_docs/python/tf/data/experimental/make_batched_features_dataset) to read and parse TFRecords data files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XW3cH1qi5oxJ"
   },
   "outputs": [],
   "source": [
    "def process_features(features, target):\n",
    "  for feature_name in CATEGORICAL_FEATURE_WITH_VOCABULARY.keys() + CATEGORICAL_FEATURE_WITH_HASH_BUCKETS.keys():\n",
    "    features[feature_name] = tf.strings.strip(features[feature_name])\n",
    "  \n",
    "  features['capital_total'] = features['capital_gain'] - features['capital_loss']\n",
    "  return features, target\n",
    "\n",
    "def make_input_fn(file_pattern, batch_size, num_epochs=1, shuffle=False):\n",
    "\n",
    "    def _input_fn():\n",
    "        dataset = tf.data.experimental.make_csv_dataset(\n",
    "            file_pattern=file_pattern,\n",
    "            batch_size=batch_size,\n",
    "            column_names=HEADER,\n",
    "            column_defaults=HEADER_DEFAULTS,\n",
    "            label_name=TARGET_NAME,\n",
    "            field_delim=',',\n",
    "            use_quote_delim=True,\n",
    "            header=False,\n",
    "            num_epochs=num_epochs,\n",
    "            shuffle=shuffle,\n",
    "            shuffle_buffer_size=(5 * batch_size),\n",
    "            shuffle_seed=RANDOM_SEED,\n",
    "            num_parallel_reads=multiprocessing.cpu_count(),\n",
    "            sloppy=True,\n",
    "        )\n",
    "        return dataset.map(process_features).cache()\n",
    "    \n",
    "    return _input_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tRYkGHG16xLU"
   },
   "outputs": [],
   "source": [
    "# You need to run tf.enable_eager_execution() at the top.\n",
    "\n",
    "dataset = make_input_fn(TRAIN_DATA_FILE, batch_size=1)()\n",
    "for features, target in dataset.take(1):\n",
    "  print \"Input Features:\"\n",
    "  for key in features:\n",
    "    print \"{}:{}\".format(key, features[key])\n",
    "\n",
    "  print \"\"\n",
    "  print \"Target:\"\n",
    "  print target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QM2SRlX0Gvk5"
   },
   "source": [
    "\n",
    "## 2. Create feature columns\n",
    "<br/>\n",
    "<img valign=\"middle\" src=\"images/tf-feature-columns.jpeg\" width=\"800\">\n",
    "\n",
    "Base feature columns\n",
    "  1. [numeric_column](https://www.tensorflow.org/api_docs/python/tf/feature_column/numeric_column|)\n",
    "  2. [categorical_column_with_vocabulary_list](https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_vocabulary_list)\n",
    "  3. [categorical_column_with_vocabulary_file](https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_vocabulary_file)\n",
    "  4. [categorical_column_with_identity](https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_identity)\n",
    "  5. [categorical_column_with_hash_buckets](https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_hash_bucket)\n",
    "\n",
    "Extended feature columns\n",
    "  1. [bucketized_column](https://www.tensorflow.org/api_docs/python/tf/feature_column/bucketized_column)\n",
    "  2. [indicator_column](https://www.tensorflow.org/api_docs/python/tf/feature_column/indicator_column)\n",
    "  3. [crossing_column](https://www.tensorflow.org/api_docs/python/tf/feature_column/crossed_column)\n",
    "  4. [embedding_column](https://www.tensorflow.org/api_docs/python/tf/feature_column/embedding_column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5cI4d7VqEadF"
   },
   "outputs": [],
   "source": [
    "def create_feature_columns():\n",
    "    \n",
    "    wide_columns = []\n",
    "    deep_columns = []\n",
    "    \n",
    "    for column in NUMERIC_FEATURE_NAMES:\n",
    "        # Create numeric columns.\n",
    "        numeric_column = tf.feature_column.numeric_column(column)\n",
    "        deep_columns.append(numeric_column)\n",
    "        \n",
    "    for column in CATEGORICAL_FEATURE_WITH_VOCABULARY:\n",
    "      # Create categorical columns with vocab.\n",
    "      vocabolary = CATEGORICAL_FEATURE_WITH_VOCABULARY[column]\n",
    "      categorical_column = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "        column, vocabolary)\n",
    "      wide_columns.append(categorical_column)\n",
    "  \n",
    "      # Create embeddings of the categorical columns.\n",
    "      embed_size = int(math.sqrt(len(vocabolary)))\n",
    "      embedding_column = tf.feature_column.embedding_column(\n",
    "        categorical_column, embed_size)\n",
    "      deep_columns.append(embedding_column)\n",
    "\n",
    "    for column in CATEGORICAL_FEATURE_WITH_HASH_BUCKETS:\n",
    "      # Create categorical columns with hashing.\n",
    "      hash_columns = tf.feature_column.categorical_column_with_hash_bucket(\n",
    "        column, \n",
    "        hash_bucket_size=CATEGORICAL_FEATURE_WITH_HASH_BUCKETS[column])\n",
    "      wide_columns.append(hash_columns)\n",
    "\n",
    "      # Create indicators for hashing columns.\n",
    "      indicator_column = tf.feature_column.indicator_column(hash_columns) \n",
    "      deep_columns.append(indicator_column)\n",
    "\n",
    "    # Create bucktized column.\n",
    "    age_bucketized = tf.feature_column.bucketized_column(\n",
    "      deep_columns[0], boundaries = [18, 25, 30, 35, 40, 45, 50, 55, 60]\n",
    "    )\n",
    "    wide_columns.append(age_bucketized)\n",
    "\n",
    "    # Create crossing column.\n",
    "    education_X_occupation = tf.feature_column.crossed_column(\n",
    "     ['education', 'workclass'], hash_bucket_size=int(1e4))\n",
    "    wide_columns.append(education_X_occupation)\n",
    "    \n",
    "    # Create embeddings for crossing column.\n",
    "    education_X_occupation_embedded = tf.feature_column.embedding_column(\n",
    "      education_X_occupation, dimension=10)\n",
    "    deep_columns.append(education_X_occupation_embedded)\n",
    "  \n",
    "    return wide_columns, deep_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 513
    },
    "colab_type": "code",
    "id": "Z7OG3ywDNIpE",
    "outputId": "814acedc-05ab-49a5-f4b7-f86434674c7d"
   },
   "outputs": [],
   "source": [
    "wide_columns, deep_columns = create_feature_columns()\n",
    "\n",
    "print \"\"\n",
    "print \"Wide columns:\"\n",
    "for column in wide_columns:\n",
    "  print column\n",
    "\n",
    "print \"\"\n",
    "print \"Deep columns:\"\n",
    "for column in deep_columns:\n",
    "  print column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BBBx7OaMPZIa"
   },
   "source": [
    "## 3. Instantiate a [Wide and Deep Estimator](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNLinearCombinedClassifier)\n",
    "<br/>\n",
    "<img valign=\"middle\" src=\"images/dnn-wide-deep.jpeg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NwdgXS-bPXlE"
   },
   "outputs": [],
   "source": [
    "def create_estimator(params, run_config):\n",
    "    \n",
    "    wide_columns, deep_columns = create_feature_columns()\n",
    "    \n",
    "    estimator = tf.estimator.DNNLinearCombinedClassifier(\n",
    "\n",
    "        n_classes=len(TARGET_LABELS),\n",
    "        label_vocabulary=TARGET_LABELS,\n",
    "        weight_column=WEIGHT_COLUMN_NAME,\n",
    "\n",
    "        dnn_feature_columns=deep_columns,\n",
    "        dnn_optimizer=tf.train.AdamOptimizer(\n",
    "          learning_rate=params.learning_rate),\n",
    "        dnn_hidden_units=params.hidden_units,\n",
    "        dnn_dropout=params.dropout,\n",
    "        dnn_activation_fn=tf.nn.relu,\n",
    "        batch_norm=True,\n",
    "\n",
    "        linear_feature_columns=wide_columns,\n",
    "        linear_optimizer='Ftrl',\n",
    "\n",
    "        config=run_config\n",
    "    )\n",
    "    \n",
    "    return estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "koqDlOh1RezR"
   },
   "source": [
    "## 4. Implement Train and Evaluate Experiment\n",
    "\n",
    "<img valign=\"middle\" src=\"images/tf-estimators.jpeg\" width=\"900\">\n",
    "\n",
    "**Delete** the **model_dir** file if you don't want a **Warm Start**\n",
    "* If not deleted, and you **change** the model, it will error.\n",
    "\n",
    "[TrainSpec](https://www.tensorflow.org/api_docs/python/tf/estimator/TrainSpec)\n",
    "* Set **shuffle** in the **input_fn** to **True**\n",
    "* Set **num_epochs** in the **input_fn** to **None**\n",
    "* Set **max_steps**. One batch (feed-forward pass & backpropagation) \n",
    "corresponds to 1 training step. \n",
    "\n",
    "[EvalSpec](https://www.tensorflow.org/api_docs/python/tf/estimator/EvalSpec)\n",
    "* Set **shuffle** in the **input_fn** to **False**\n",
    "* Set Set **num_epochs** in the **input_fn** to **1**\n",
    "* Set **steps** to **None** if you want to use all the evaluation data. \n",
    "* Otherwise, set **steps** to the number of batches you want to use for evaluation, and set **shuffle** to True.\n",
    "* Set **start_delay_secs** to 0  to start evaluation as soon as a checkpoint is produced.\n",
    "* Set **throttle_secs** to 0 to re-evaluate as soon as a new checkpoint is produced.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kcAvhhtlOTCd"
   },
   "outputs": [],
   "source": [
    "def run_experiment(estimator, params, run_config, \n",
    "  resume=False, train_hooks=None, exporters=None):\n",
    "\n",
    "  print \"Resume training {}: \".format(resume)\n",
    "  print \"Epochs: {}\".format(epochs)\n",
    "  print \"Batch size: {}\".format(params.batch_size)\n",
    "  print \"Steps per epoch: {}\".format(steps_per_epoch)\n",
    "  print \"Training steps: {}\".format(params.max_steps)\n",
    "  print \"Learning rate: {}\".format(params.learning_rate)\n",
    "  print \"Hidden Units: {}\".format(params.hidden_units)\n",
    "  print \"Dropout probability: {}\".format(params.dropout)\n",
    "  print \"Save a checkpoint and evaluate afer {} step(s)\".format(run_config.save_checkpoints_steps)\n",
    "  print \"Keep the last {} checkpoint(s)\".format(run_config.keep_checkpoint_max)\n",
    "  print \"\"\n",
    "  \n",
    "  tf.logging.set_verbosity(tf.logging.INFO)\n",
    "\n",
    "  if not resume: \n",
    "    if tf.gfile.Exists(run_config.model_dir):\n",
    "      print \"Removing previous artefacts...\"\n",
    "      tf.gfile.DeleteRecursively(run_config.model_dir)\n",
    "  else:\n",
    "    print \"Resuming training...\"\n",
    "\n",
    "  # Create train specs.\n",
    "  train_spec = tf.estimator.TrainSpec(\n",
    "      input_fn = make_input_fn(\n",
    "          TRAIN_DATA_FILE,\n",
    "          batch_size=params.batch_size,\n",
    "          num_epochs=None, # Run until the max_steps is reached.\n",
    "          shuffle=True\n",
    "      ),\n",
    "      max_steps=params.max_steps,\n",
    "      hooks=train_hooks\n",
    "  )\n",
    "\n",
    "  # Create eval specs.\n",
    "  eval_spec = tf.estimator.EvalSpec(\n",
    "      input_fn = make_input_fn(\n",
    "          EVAL_DATA_FILE,\n",
    "          batch_size=params.batch_size,     \n",
    "      ),\n",
    "      exporters=exporters,\n",
    "      start_delay_secs=0,\n",
    "      throttle_secs=0,\n",
    "      steps=None # Set to limit number of steps for evaluation.\n",
    "  )\n",
    "  \n",
    "  time_start = datetime.utcnow() \n",
    "  print \"Experiment started at {}\".format(time_start.strftime(\"%H:%M:%S\"))\n",
    "  print \".......................................\"\n",
    "  \n",
    "  # Run train and evaluate.\n",
    "  tf.estimator.train_and_evaluate(\n",
    "    estimator=estimator,\n",
    "    train_spec=train_spec, \n",
    "    eval_spec=eval_spec)\n",
    "\n",
    "  time_end = datetime.utcnow() \n",
    "  print \".......................................\"\n",
    "  print \"Experiment finished at {}\".format(time_end.strftime(\"%H:%M:%S\"))\n",
    "  print \"\"\n",
    "  \n",
    "  time_elapsed = time_end - time_start\n",
    "  print \"Experiment elapsed time: {} seconds\".format(time_elapsed.total_seconds())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1Khy9RYnfph4"
   },
   "source": [
    "## Set Parameters and Run Configurations.\n",
    "\n",
    "\n",
    "* Set **model_dir** in the **run_config**\n",
    "* If the data **size is known**, training **steps**, with respect to **epochs** would be: **(training_size / batch_size) * epochs** \n",
    "* By default, a **checkpoint** is saved every 600 secs.  That is, the model is **evaluated** only every 10mins. \n",
    "* To change this behaviour, set one of the following parameters in the **run_config**\n",
    " * **save_checkpoints_secs**: Save checkpoints every this many **seconds**.\n",
    " * **save_checkpoints_steps**: Save checkpoints every this many **steps**.\n",
    "\n",
    "* Set the number of the checkpoints to keep using **keep_checkpoint_max**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GUyIbhvBRWqV"
   },
   "outputs": [],
   "source": [
    "class Parameters():\n",
    "  pass\n",
    "\n",
    "MODELS_LOCATION = 'gs://ksalama-gcs-cloudml/others/models/census'\n",
    "MODEL_NAME = 'dnn_classifier'\n",
    "model_dir = os.path.join(MODELS_LOCATION, MODEL_NAME)\n",
    "os.environ['MODEL_DIR'] = model_dir\n",
    "\n",
    "TRAIN_DATA_SIZE = 32561\n",
    "\n",
    "params = Parameters()\n",
    "params.learning_rate = 0.001\n",
    "params.hidden_units = [128, 128, 128]\n",
    "params.dropout = 0.15\n",
    "params.batch_size =  128\n",
    "\n",
    "# Set number of steps with respect to epochs.\n",
    "epochs = 5\n",
    "steps_per_epoch = int(math.ceil(TRAIN_DATA_SIZE / params.batch_size))\n",
    "params.max_steps = steps_per_epoch * epochs\n",
    "\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    tf_random_seed=RANDOM_SEED,\n",
    "    save_checkpoints_steps=steps_per_epoch, # Save a checkpoint after each epoch, evaluate the model after each epoch.\n",
    "    keep_checkpoint_max=3, # Keep the 3 most recently  produced checkpoints.\n",
    "    model_dir=model_dir,\n",
    "    save_summary_steps=100, # Summary steps for Tensorboard.\n",
    "    log_step_count_steps=50\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LyvxkYMvtVUC"
   },
   "source": [
    "## Run Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "KyG4bqn5zerp",
    "outputId": "66fa22cb-4259-46a0-936c-71837f1bc6aa"
   },
   "outputs": [],
   "source": [
    "if COLAB:\n",
    "  from tensorboardcolab import *\n",
    "  TensorBoardColab(graph_path=model_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1040
    },
    "colab_type": "code",
    "id": "APPdvnMRtVZi",
    "outputId": "9211edf0-631d-4850-987d-e400563f7279"
   },
   "outputs": [],
   "source": [
    "estimator = create_estimator(params, run_config)\n",
    "run_experiment(estimator, params, run_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "id": "gBsemiGotiZ-",
    "outputId": "5cea834f-31bb-4afd-b62c-e00084736fbd"
   },
   "outputs": [],
   "source": [
    "print model_dir\n",
    "!gcloud storage ls {model_dir}"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iYrBHd6fDRSI"
   },
   "source": [
    "## 5. Export your trained model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gz3jaXq8LQfc"
   },
   "source": [
    "### Implement serving input receiver function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SZm-eqneDQ3M"
   },
   "outputs": [],
   "source": [
    "def make_serving_input_receiver_fn():\n",
    "  inputs = {}\n",
    "  for feature_name in FEATURE_NAMES:\n",
    "    dtype = tf.float32 if feature_name in NUMERIC_FEATURE_NAMES else tf.string\n",
    "    inputs[feature_name] = tf.placeholder(shape=[None], dtype=dtype)\n",
    "  \n",
    "  # What is wrong here? \n",
    "  return tf.estimator.export.build_raw_serving_input_receiver_fn(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rnz1N_oQLbWj"
   },
   "source": [
    "### Export to saved_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fAqNSl9uLbmn"
   },
   "outputs": [],
   "source": [
    "export_dir = os.path.join(model_dir, 'export')\n",
    "\n",
    "# Delete export directory if exists.\n",
    "if tf.gfile.Exists(export_dir):\n",
    "    tf.gfile.DeleteRecursively(export_dir)\n",
    "\n",
    "# Export the estimator as a saved_model.\n",
    "estimator.export_savedmodel(\n",
    "    export_dir_base=export_dir,\n",
    "    serving_input_receiver_fn=make_serving_input_receiver_fn()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "ZFsxiySNzltk",
    "outputId": "82019105-912a-4939-9fa8-79e55da04d3a"
   },
   "outputs": [],
   "source": [
    "!gcloud storage ls gs://ksalama-gcs-cloudml/others/models/census/dnn_classifier/export/1552582374"   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1428
    },
    "colab_type": "code",
    "id": "-JO1gunHH6r7",
    "outputId": "2eb64e19-29ff-40ba-ed17-729791035733"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "saved_models_base=${MODEL_DIR}/export/\n",
    "saved_model_dir=$(gcloud storage ls ${saved_models_base} | tail -n 1)\n",    "saved_model_cli show --dir=${saved_model_dir} --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "B-kEFV1PLJXs"
   },
   "source": [
    "### Test saved_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "id": "2pN4u67aIF9A",
    "outputId": "d053425f-3014-4b2d-f12a-8f4667029039"
   },
   "outputs": [],
   "source": [
    "export_dir = os.path.join(model_dir, 'export')\n",
    "tf.gfile.ListDirectory(export_dir)[-1]\n",
    "saved_model_dir = os.path.join(export_dir, tf.gfile.ListDirectory(export_dir)[-1])\n",
    "print(saved_model_dir)\n",
    "print \"\"\n",
    "\n",
    "predictor_fn = tf.contrib.predictor.from_saved_model(\n",
    "    export_dir = saved_model_dir,\n",
    "    signature_def_key=\"predict\"\n",
    ")\n",
    "\n",
    "output = predictor_fn(\n",
    "    {\n",
    "        'age': [34.0],\n",
    "        'workclass': ['Private'],\n",
    "        'education': ['Doctorate'],\n",
    "        'education_num': [10.0],\n",
    "        'marital_status': ['Married-civ-spouse'],\n",
    "        'occupation': ['Prof-specialty'],\n",
    "        'relationship': ['Husband'],\n",
    "        'race': ['White'],\n",
    "        'gender': ['Male'],\n",
    "        'capital_gain': [0.0], \n",
    "        'capital_loss': [0.0], \n",
    "        'hours_per_week': [40.0],\n",
    "        'native_country':['Egyptian']\n",
    "    }\n",
    ")\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0kOugLPBLvSR"
   },
   "source": [
    "## Export the Model during Training and Evaluation\n",
    "Saved models are exported under **<model_dir>/export/<folder_name>**.\n",
    "* [Latest Exporter](https://www.tensorflow.org/api_docs/python/tf/estimator/LatestExporter): exports a model **after each evaluation**.\n",
    " * specify the **maximum number** of exported models to keep using **exports_to_keep** param.\n",
    "* [Final Exporter](https://www.tensorflow.org/api_docs/python/tf/estimator/LatestExporter): exports only the very **last** evaluated checkpoint. of the model.\n",
    "* [Best exporter](https://www.tensorflow.org/api_docs/python/tf/estimator/BestExporter): runs everytime when the **newly evaluted checkpoint** is **better** than any exsiting model.\n",
    " * specify the **maximum number** of exported models to keep using **exports_to_keep** param.\n",
    " * It uses the **evaluation events** stored under the **eval** folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 3590
    },
    "colab_type": "code",
    "id": "TXms_p48NV46",
    "outputId": "e41fa186-ab81-4361-ee89-af3f27b87d0b"
   },
   "outputs": [],
   "source": [
    "def _accuracy_bigger(best_eval_result, current_eval_result):\n",
    "  \n",
    "  metric = 'accuracy'\n",
    "  return best_eval_result[metric] < current_eval_result[metric]\n",
    "\n",
    "\n",
    "params.max_steps = 1000\n",
    "params.hidden_units = [128, 128]\n",
    "params.dropout = 0\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    tf_random_seed=RANDOM_SEED,\n",
    "    save_checkpoints_steps=200,\n",
    "    keep_checkpoint_max=1,\n",
    "    model_dir=model_dir,\n",
    "    log_step_count_steps=50\n",
    ")\n",
    "\n",
    "exporter = tf.estimator.BestExporter(\n",
    "    compare_fn=_accuracy_bigger,\n",
    "    event_file_pattern='eval_{}/*.tfevents.*'.format(datetime.utcnow().strftime(\"%H%M%S\")),\n",
    "    name=\"estimate\", # Saved models are exported under /export/estimate/\n",
    "    serving_input_receiver_fn=make_serving_input_receiver_fn(),\n",
    "    exports_to_keep=1\n",
    ")\n",
    "\n",
    "estimator = create_estimator(params, run_config)\n",
    "run_experiment(estimator, params, run_config, exporters = [exporter])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "4BG6zREARkzd",
    "outputId": "72f03e5c-1c4c-4938-ac68-75e7e77760c2"
   },
   "outputs": [],
   "source": [
    "!gcloud storage ls {model_dir}/export/estimate"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ANWtVZMkWB07"
   },
   "source": [
    "## 6. Early Stopping\n",
    "* [stop_if_higher_hook](https://www.google.com/search?q=stop_if_higher_hook&oq=stop_if_higher_hook) \n",
    "* [stop_if_lower_hook](https://www.tensorflow.org/api_docs/python/tf/contrib/estimator/stop_if_lower_hook) \n",
    "* [stop_if_no_increase_hook](https://www.tensorflow.org/api_docs/python/tf/contrib/estimator/stop_if_no_increase_hook)\n",
    "* [stop_if_no_decrease_hook](https://www.tensorflow.org/api_docs/python/tf/contrib/estimator/stop_if_no_decrease_hook)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1584
    },
    "colab_type": "code",
    "id": "kNEUjWciT3b0",
    "outputId": "e5e34d19-1073-4d9a-eec6-d76798f4200e"
   },
   "outputs": [],
   "source": [
    "early_stopping_hook = tf.contrib.estimator.stop_if_no_increase_hook(\n",
    "    estimator,\n",
    "    'accuracy',\n",
    "    max_steps_without_increase=100,\n",
    "    run_every_secs=None,\n",
    "    run_every_steps=500\n",
    ")\n",
    "\n",
    "params.max_steps = 1000000\n",
    "params.hidden_units = [128, 128]\n",
    "params.dropout = 0\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    tf_random_seed=RANDOM_SEED,\n",
    "    save_checkpoints_steps=500,\n",
    "    keep_checkpoint_max=1,\n",
    "    model_dir=model_dir,\n",
    "    log_step_count_steps=100\n",
    ")\n",
    "\n",
    "run_experiment(estimator, params, run_config, exporters = [exporter], train_hooks=[early_stopping_hook])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PZx3xX6hZuNJ"
   },
   "source": [
    "## 7. Using Distribution Strategy for Utilising Multiple GPUs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zRt7d2o7Zs8f"
   },
   "outputs": [],
   "source": [
    "strategy = None\n",
    "num_gpus = len([device_name for device_name in tf.contrib.eager.list_devices()\n",
    "                if '/device:GPU' in device_name])\n",
    "\n",
    "print \"GPUs available: {}\".format(num_gpus)\n",
    "\n",
    "if num_gpus > 1:\n",
    "    strategy = tf.distribute.MirroredStrategy()\n",
    "    params.batch_size = int(math.ceil(params.batch_size / num_gpus))\n",
    "\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    tf_random_seed=RANDOM_SEED,\n",
    "    save_checkpoints_steps=200,\n",
    "    model_dir=model_dir,\n",
    "    train_distribute=strategy\n",
    ")\n",
    "\n",
    "estimator = create_estimator(params, run_config)\n",
    "run_experiment(estimator, params, run_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2Jc0FsSpcxSD"
   },
   "source": [
    "## 8. Extending a Premade Estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KMzijRO-c_2_"
   },
   "source": [
    "### Add an evaluation metric\n",
    "* [tf.metrics](https://www.tensorflow.org/api_docs/python/tf/metrics)\n",
    "* [tf.estimator.add_metric](https://www.tensorflow.org/api_docs/python/tf/contrib/estimator/add_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1074
    },
    "colab_type": "code",
    "id": "C-1h1WYkY8FJ",
    "outputId": "961b6911-e9d6-4c1c-92c3-598ec2585edb"
   },
   "outputs": [],
   "source": [
    "def metric_fn(labels, predictions):\n",
    "\n",
    "  metrics = {}\n",
    "  \n",
    "  label_index = tf.contrib.lookup.index_table_from_tensor(tf.constant(TARGET_LABELS)).lookup(labels)\n",
    "  one_hot_labels = tf.one_hot(label_index, len(TARGET_LABELS))\n",
    "\n",
    "  metrics['mirco_accuracy'] = tf.metrics.mean_per_class_accuracy(\n",
    "    labels=label_index,\n",
    "    predictions=predictions['class_ids'],\n",
    "    num_classes=2)\n",
    "  \n",
    "  metrics['f1_score'] = tf.contrib.metrics.f1_score(\n",
    "    labels=one_hot_labels,\n",
    "    predictions=predictions['probabilities'])\n",
    "\n",
    "  return metrics\n",
    "\n",
    "params.max_steps = 1\n",
    "estimator = create_estimator(params, run_config)\n",
    "estimator = tf.contrib.estimator.add_metrics(estimator, metric_fn)\n",
    "run_experiment(estimator, params, run_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p7KfzoD2j_JE"
   },
   "source": [
    "### Add Forward Features\n",
    "* [tf.estimator.forward_features](https://www.tensorflow.org/api_docs/python/tf/contrib/estimator/forward_features)\n",
    "This is very useful for batch prediction, in order to make instances to their predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 156
    },
    "colab_type": "code",
    "id": "GOn0O0w9eXzm",
    "outputId": "acbab984-73ef-470d-a15d-f2992ec810ac"
   },
   "outputs": [],
   "source": [
    "estimator = tf.contrib.estimator.forward_features(estimator, keys=\"row_identifier\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 445
    },
    "colab_type": "code",
    "id": "ERHxBSHsklWw",
    "outputId": "80bd10b9-ff41-4ae6-882d-df0f05726a09"
   },
   "outputs": [],
   "source": [
    "def make_serving_input_receiver_fn():\n",
    "  inputs = {}\n",
    "  for feature_name in FEATURE_NAMES:\n",
    "    dtype = tf.float32 if feature_name in NUMERIC_FEATURE_NAMES else tf.string\n",
    "    inputs[feature_name] = tf.placeholder(shape=[None], dtype=dtype)\n",
    "  \n",
    "  processed_inputs,_ = process_features(inputs, None)\n",
    "  processed_inputs['row_identifier'] = tf.placeholder(shape=[None], dtype=tf.string)\n",
    "  return tf.estimator.export.build_raw_serving_input_receiver_fn(processed_inputs)\n",
    "\n",
    "export_dir = os.path.join(model_dir, 'export')\n",
    "\n",
    "if tf.gfile.Exists(export_dir):\n",
    "    tf.gfile.DeleteRecursively(export_dir)\n",
    "        \n",
    "estimator.export_savedmodel(\n",
    "    export_dir_base=export_dir,\n",
    "    serving_input_receiver_fn=make_serving_input_receiver_fn()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1564
    },
    "colab_type": "code",
    "id": "CTvGsBEBksR0",
    "outputId": "86b87a08-d6e1-42da-a564-993ad73ff31d"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "saved_models_base=${MODEL_DIR}/export/\n",
    "saved_model_dir=$(gcloud storage ls ${saved_models_base} | tail -n 1)\n",    "saved_model_cli show --dir=${saved_model_dir} --all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "id": "J0xhpPMzmHLG",
    "outputId": "aa8128ae-6dfa-4647-e135-574e99557c1f"
   },
   "outputs": [],
   "source": [
    "export_dir = os.path.join(model_dir, 'export')\n",
    "tf.gfile.ListDirectory(export_dir)[-1]\n",
    "saved_model_dir = os.path.join(export_dir, tf.gfile.ListDirectory(export_dir)[-1])\n",
    "print(saved_model_dir)\n",
    "print \"\"\n",
    "\n",
    "predictor_fn = tf.contrib.predictor.from_saved_model(\n",
    "    export_dir = saved_model_dir,\n",
    "    signature_def_key=\"predict\"\n",
    ")\n",
    "\n",
    "output = predictor_fn(\n",
    "    {   'row_identifier': ['key0123'],\n",
    "        'age': [34.0],\n",
    "        'workclass': ['Private'],\n",
    "        'education': ['Doctorate'],\n",
    "        'education_num': [10.0],\n",
    "        'marital_status': ['Married-civ-spouse'],\n",
    "        'occupation': ['Prof-specialty'],\n",
    "        'relationship': ['Husband'],\n",
    "        'race': ['White'],\n",
    "        'gender': ['Male'],\n",
    "        'capital_gain': [0.0], \n",
    "        'capital_loss': [0.0], \n",
    "        'hours_per_week': [40.0],\n",
    "        'native_country':['Egyptian']\n",
    "    }\n",
    ")\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QVfpA-NN3o5G"
   },
   "source": [
    "## 9. Adaptive learning rate\n",
    "* [exponential_decay](https://www.tensorflow.org/api_docs/python/tf/train/exponential_decay)\n",
    "* [consine_decay](https://www.tensorflow.org/api_docs/python/tf/train/cosine_decay)\n",
    "* [linear_cosine_decay](https://www.tensorflow.org/api_docs/python/tf/train/linear_cosine_decay)\n",
    "* [consine_decay_restarts](https://www.tensorflow.org/api_docs/python/tf/train/cosine_decay_restarts)\n",
    "* [polynomial decay](https://www.tensorflow.org/api_docs/python/tf/train/polynomial_decay)\n",
    "* [piecewise_constant_decay](https://www.tensorflow.org/api_docs/python/tf/train/piecewise_constant_decay)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "N_N9HocamU5b"
   },
   "outputs": [],
   "source": [
    "def create_estimator(params, run_config):\n",
    "  \n",
    "  wide_columns, deep_columns = create_feature_columns()\n",
    "  \n",
    "  def _update_optimizer(initial_learning_rate, decay_steps):\n",
    "    \n",
    "    # learning_rate = tf.train.exponential_decay(\n",
    "    #   initial_learning_rate,\n",
    "    #   global_step=tf.train.get_global_step(),\n",
    "    #   decay_steps=decay_steps,\n",
    "    #   decay_rate=0.9\n",
    "    # )\n",
    "\n",
    "    learning_rate = tf.train.cosine_decay_restarts(\n",
    "      initial_learning_rate,\n",
    "      tf.train.get_global_step(),\n",
    "      first_decay_steps=50,\n",
    "      t_mul=2.0,\n",
    "      m_mul=1.0,\n",
    "      alpha=0.0,\n",
    "    )\n",
    "\n",
    "    tf.summary.scalar('learning_rate', learning_rate)\n",
    "\n",
    "    return tf.train.AdamOptimizer(\n",
    "      learning_rate=initial_learning_rate)\n",
    "    \n",
    "  estimator = tf.estimator.DNNLinearCombinedClassifier(\n",
    "\n",
    "    n_classes=len(TARGET_LABELS),\n",
    "    label_vocabulary=TARGET_LABELS,\n",
    "    weight_column=WEIGHT_COLUMN_NAME,\n",
    "\n",
    "    dnn_feature_columns=deep_columns,\n",
    "    dnn_optimizer=lambda: _update_optimizer(params.learning_rate, params.max_steps),\n",
    "    dnn_hidden_units=params.hidden_units,\n",
    "    dnn_dropout=params.dropout,\n",
    "    batch_norm=True,\n",
    "    \n",
    "    linear_feature_columns=wide_columns,\n",
    "    linear_optimizer='Ftrl',\n",
    "    \n",
    "    config=run_config\n",
    "  )\n",
    "  \n",
    "  return estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 2128
    },
    "colab_type": "code",
    "id": "oSZLQLaz4FQ_",
    "outputId": "c2cee42e-cb8f-4ed8-e4a8-b01ae45981f6"
   },
   "outputs": [],
   "source": [
    "params.learning_rate = 0.1\n",
    "params.max_steps = 1000\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    tf_random_seed=RANDOM_SEED,\n",
    "    save_checkpoints_steps=200,\n",
    "    model_dir=model_dir,\n",
    ")\n",
    "\n",
    "if COLAB:\n",
    "  from tensorboardcolab import *\n",
    "  TensorBoardColab(graph_path=model_dir)\n",
    "\n",
    "estimator = create_estimator(params, run_config)\n",
    "run_experiment(estimator, params, run_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x1bahMKz848A"
   },
   "source": [
    "## License"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "q5hU3tDV85FP"
   },
   "source": [
    "Author: Khalid Salama\n",
    "\n",
    "---\n",
    "**Disclaimer**: This is not an official Google product. The sample code provided for an educational purpose.\n",
    "\n",
    "---\n",
    "\n",
    "Copyright 2019 Google LLC\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "0kOugLPBLvSR",
    "ANWtVZMkWB07",
    "PZx3xX6hZuNJ",
    "2Jc0FsSpcxSD",
    "KMzijRO-c_2_",
    "p7KfzoD2j_JE",
    "QVfpA-NN3o5G"
   ],
   "name": "01-tf_estimator_deepdive.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
